Method and system for digital staining of label-free fluorescence images using deep learning

ABSTRACT

A deep learning-based digital staining method and system are disclosed that enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples based on autofluorescence images acquired using a fluorescent microscope. The system and method have particular applicability for the creation of digitally/virtually-stained whole slide images (WSIs) of unlabeled/unstained tissue samples that are analyzes by a histopathologist. The methods bypass the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses, in one embodiment, a convolutional neural network trained using a generative adversarial network model to transform fluorescence images of an unlabeled sample into an image that is equivalent to the brightfield image of the chemically stained-version of the same sample. This label-free digital staining method eliminates cumbersome and costly histochemical staining procedures and significantly simplifies tissue preparation in pathology and histology fields.

RELATED APPLICATION

This Application is a U.S. National Stage filing under 35 U.S.C. § 371of International Application No. PCT/US2019/025020, filed Mar. 29, 2019,which claims priority to U.S. Provisional Patent Application No.62/651,005 filed on Mar. 30, 2018, which are hereby incorporated byreference. Priority is claimed pursuant to 35 U.S.C. §§ 119, 371 and anyother applicable statute.

TECHNICAL FIELD

The technical field generally relates to methods and systems used toimage unstained (i.e., label-free) tissue. In particular, the technicalfield relates to microscopy methods and systems that utilize deep neuralnetwork learning for digitally or virtually staining of images ofunstained or unlabeled tissue. Deep learning in neural networks, a classof machine learning algorithms, are used to digitally stain images oflabel-free tissue sections into images that are equivalent to microscopyimages of the same samples that are stained or labelled.

BACKGROUND

Microscopic imaging of tissue samples is a fundamental tool used for thediagnosis of various diseases and forms the workhorse of pathology andbiological sciences. The clinically-established gold standard image of atissue section is the result of a laborious process, which includes thetissue specimen being formalin-fixed paraffin-embedded (FFPE), sectionedto thin slices (typically ˜2-10 μm), labeled/stained and mounted on aglass slide, which is then followed by its microscopic imaging usinge.g., a brightfield microscope. All these steps use multiple reagentsand introduce irreversible effects on the tissue. There have been recentefforts to change this workflow using different imaging modalities.Attempts have been made to imaged fresh, non-paraffin-embedded tissuesamples using non-linear microscopy methods based on e.g., two-photonfluorescence, second harmonic generation, third-harmonic generation aswell as Raman scattering. Other attempts have used a controllablesuper-continuum source to acquire multi-modal images for chemicalanalysis of fresh tissue samples. These methods require using ultra-fastlasers or super-continuum sources, which might not be readily availablein most settings and require relatively long scanning times due toweaker optical signals. In addition to these, other microscopy methodsfor imaging non-sectioned tissue samples have also emerged by usingUV-excitation on stained samples, or by taking advantage of thefluorescence emission of biological tissue at short wavelengths.

In fact, fluorescence signal creates some unique opportunities forimaging tissue samples by making use of the fluorescent light emittedfrom endogenous fluorophores. It has been demonstrated that suchendogenous fluorescence signatures carry useful information that can bemapped to functional and structural properties of biological specimenand therefore have been used extensively for diagnostics and researchpurposes. One of the main focus areas of these efforts has been thespectroscopic investigation of the relationship between differentbiological molecules and their structural properties under differentconditions. Some of these well-characterized biological constituentsinclude vitamins (e.g., vitamin A, riboflavin, thiamin), collagen,coenzymes, fatty acids, among others.

While some of the above discussed techniques have unique capabilities todiscriminate e.g., cell types and sub-cellular components in tissuesamples using various contrast mechanisms, pathologists as well as tumorclassification software are in general trained for examining “goldstandard” stained tissue samples to make diagnostic decisions. Partiallymotivated by this, some of the above-mentioned techniques have beenaugmented to create pseudo-Hematoxylin and Eosin (H&E) images, which arebased on a linear approximation that relates the fluorescence intensityof an image to the dye concentration per tissue volume, usingempirically determined constants that represent the mean spectralresponse of various dyes embedded in the tissue. These methods also usedexogenous staining to enhance the fluorescence signal contrast in orderto create virtual H&E images of tissue samples.

SUMMARY

In one embodiment, a system and method are provided that utilizes atrained deep neural network that is used for the digital or virtualstaining of label-free thin tissue sections or other samples using theirfluorescence images obtained from chemically unstained tissue (or othersamples). Chemically unstained tissue refers to the lack of standardstains or labels used in histochemical staining of tissue. Thefluorescence of chemically unstained tissue may includeauto-fluorescence of tissue from naturally occurring or endogenousfluorophores or other endogenous emitters of light at frequenciesdifferent from the illumination frequency (i.e., frequency-shiftedlight). Fluorescence of chemically unstained tissue may further includefluorescence of tissue from exogenously added fluorescent labels orother exogenous emitters of light. Samples are imaged with afluorescence microscope such as a wide-field fluorescence microscope (ora standard fluorescence microscope). The microscope may utilize astandard near-UV excitation/emission filter set or otherexcitation/emission light source/filter sets that are known to thoseskilled in the art. The digital or virtual staining is performed, insome embodiments, on a single fluorescence image obtained of the sampleby using, in on preferred embodiment, a trained deep neural network.

In one embodiment, the trained deep neural network is a ConvolutionalNeural Network (CNN) which is trained using a Generative AdversarialNetworks (GAN) model to match the corresponding brightfield microscopicimages of tissue samples after they are labeled with a certain histologystain. In this embodiment, a fluorescence image of the unstained sample(e.g., tissue) is input to the trained deep neural network to generatethe digitally stained image. Therefore, in this embodiment, thehistochemical staining and brightfield imaging steps are completelyreplaced by the use of the trained deep neural network which generatesthe digitally stained image. As explained herein, the network inferenceperformed by the trained neural network is fast, taking in someembodiments, less than a second using a standard desktop computer for animaging field-of-view of ˜0.33 mm×0.33 mm using e.g., a 40× objectivelens. Using a 20× objective for scanning tissue, a network inferencetime of 1.9 seconds/mm² was achieved.

The deep learning-based digital/virtual histology staining method usingauto-fluorescence has been demonstrated by imaging label-free humantissue samples including salivary gland, thyroid, kidney, liver, lungand skin, where the trained deep neural network output createdequivalent images, substantially matching with the images of the samesamples that were labeled with three different stains, i.e., H&E(salivary gland and thyroid), Jones stain (kidney) and Masson'sTrichrome (liver and lung). Because the trained deep neural network'sinput image is captured by a conventional fluorescence microscope with astandard filter set, this approach has transformative potential to useunstained tissue samples for pathology and histology applications,entirely bypassing the histochemical staining process, saving time andthe attendant costs. This includes the cost of labor, reagents, theadditional time involved in staining processes, and the like. Forexample, for the histology stains that were approximated using thedigital or virtual staining process described herein, each stainingprocedure of a tissue section on average takes ˜45 min (H&E) and 2-3hours (Masson's Trichrome and Jones stain), with an estimated cost,including labor, of $2-5 for H&E and >$16-35 for Masson's Trichrome andJones stain. Furthermore, some of these histochemical staining processesrequire time-sensitive steps, demanding the expert to monitor theprocess under a microscope, which makes the entire process not onlylengthy and relatively costly, but also laborious. The system and methoddisclosed herein bypasses all these staining steps, and also allows thepreservation of unlabeled tissue sections for later analysis, such asmicro-marking of sub-regions of interest on the unstained tissuespecimen that can be used for more advanced immunochemical and molecularanalysis to facilitate e.g., customized therapies. Furthermore, thestaining efficacy of this approach for whole slide images (WSIs)corresponding to some of these samples was blindly evaluated by a groupof pathologists, who were able to recognize histopathological featureswith the digital/virtual staining technique, achieving a high degree ofagreement with the histologically stained images of the same samples.

Further, this deep learning-based digital/virtual histology stainingframework can be broadly applied to other excitation wavelengths orfluorescence filter sets, as well as to other microscopy modalities(such as non-linear microscopy) that utilize additional endogenouscontrast mechanisms. In the experiments, sectioned and fixed tissuesamples were used to be able to provide meaningful comparisons to theresults of the standard histochemical staining process. However, thepresented approach would also work with non-fixed, non-sectioned tissuesamples, potentially making it applicable to use in surgery rooms or atthe site of a biopsy for rapid diagnosis or telepathology applications.Beyond its clinical applications, this method could broadly benefithistology field and its applications in life science research andeducation.

In one embodiment, a method of generating a digitally stainedmicroscopic image of a label-free sample includes providing a trained,deep neural network that is run using image processing software executedusing one or more processors of a computing device, wherein the trained,deep neural network is trained with a plurality of matched chemicallystained images or image patches and their corresponding fluorescenceimages or image patches of the same sample. The label-free sample mayinclude tissues, cells, pathogens, biological fluid smears, or othermicro-objects of interest. In some embodiments, the deep neural networkmay be trained using one or more tissue type/chemical stain typecombinations. For example, this may include tissue type A with stain #1,stain #2, stain #3, etc. In some embodiments, the deep neural networkmay be trained using tissue that has been stained with multiple stains.

A fluorescence image of the sample is input to the trained, deep neuralnetwork. The trained, deep neural network then outputs a digitallystained microscopic image of the sample based on the input fluorescenceimage of the sample. In one embodiment, the trained, deep neural networkis a convolutional neural network (CNN). This may include a CNN thatuses a Generative Adversarial Network (GAN) model. The fluorescenceinput image of the sample is obtained using a fluorescence microscopeand an excitation light source (e.g., UV or near UV emitting lightsource). In some alternative embodiments, multiple fluorescence imagesare input into the trained, deep neural network. For example, onefluorescence image may be obtained at a first filtered wavelength orwavelength range while another fluorescence image may be obtained at asecond filtered wavelength or wavelength range. These two fluorescenceimages are then input into the trained, deep neural network to output asingle digitally/virtually stained image. In another embodiment, theobtained fluorescence image may be subject to one or more linear ornon-linear pre-processing operations selected from contrast enhancement,contrast reversal, image filtering which may be input alone or incombination with the obtained fluorescence image into the trained, deepneural network.

For example, in another embodiment, a method of generating a digitallystained microscopic image of a label-free sample includes providing atrained, deep neural network that is executed by image processingsoftware using one or more processors of a computing device, wherein thetrained, deep neural network is trained with a plurality of matchedchemically stained images or image patches and their correspondingfluorescence images or image patches of the same sample. A firstfluorescence image of the sample is obtained using a fluorescencemicroscope and wherein fluorescent light at a first emission wavelengthor wavelength range is emitted from endogenous fluorophores or otherendogenous emitters of frequency-shifted light within the sample. Asecond fluorescence image of the sample is obtained using a fluorescencemicroscope and wherein fluorescent light at a second emission wavelengthor wavelength range is emitted from endogenous fluorophores or otherendogenous emitters of frequency-shifted light within the sample. Thefirst and second fluorescence images may be obtained by using differentexcitation/emission wavelength combinations. The first and secondfluorescence images of the sample are then input to the trained, deepneural network, the trained, deep neural network outputting thedigitally stained microscopic image of the sample that is substantiallyequivalent to a corresponding brightfield image of the same sample thathas been chemically stained.

In another embodiment, a system for generating digitally stainedmicroscopic images of a chemically unstained sample includes a computingdevice having image processing software executed thereon or thereby, theimage processing software comprising a trained, deep neural network thatis executed using one or more processors of the computing device. Thetrained, deep neural network is trained with a plurality of matchedchemically stained images or image patches and their correspondingfluorescence images or image patches of the same sample. The imageprocessing software is configured to receive one or more fluorescenceimage(s) of the sample and output the digitally stained microscopicimage of the sample that is substantially equivalent to a correspondingbrightfield image of the same sample that has been chemically stained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates a system that is used to generate adigitally/virtually stained output image of a sample from an unstainedmicroscope image of the sample according to one embodiment.

FIG. 2 illustrates a schematic representation of the deep learning-baseddigital/virtual histology staining operations using a fluorescence imageof unstained tissue.

FIGS. 3A-3H illustrate digital/virtual staining results that match thechemically stained H&E samples. The first two (2) columns (FIGS. 3A and3E) show the auto-fluorescence images of unstained salivary gland tissuesections (used as input to the deep neural network), and the thirdcolumn (FIGS. 3C and 3G) shows the digital/virtual staining results. Thelast column (FIGS. 3D and 3H) shows the brightfield images of the sametissue sections, after the histochemical staining process. Evaluation ofboth FIG. 3C and FIG. 3D demonstrates a small island of infiltratingtumor cells within subcutaneous fibro-adipose tissue. Note that thenuclear detail, including distinction of nucleoli (arrows in 3C and 3D)and chromatin texture, is clearly appreciated in both panels. Similarly,in FIGS. 3G and 3H the H&E stains demonstrate infiltrating squamous cellcarcinoma. The desmoplastic reaction with edematous myxoid change(asterisk in FIGS. 3G and 3H) in the adjacent stroma is clearlyidentifiable in both stains/panels.

FIGS. 4A-4H illustrate digital/virtual staining results to match thechemically stained Jones samples. The first two (2) columns (FIGS. 4A,4E) show the auto-fluorescence images of unstained kidney tissuesections (used as input to the deep neural network), and the thirdcolumn (FIGS. 4C and 4G), shows the digital/virtual staining results.The last column (FIGS. 4D, 4H) shows the brightfield images of the sametissue sections, after the histochemical staining process.

FIGS. 5A-5P illustrate digital/virtual staining results to match theMasson's Trichrome stain for liver and lung tissue sections. The firsttwo (2) columns show the auto-fluorescence images of an unstained livertissue section (rows 1 and 2—FIGS. 5A, 5B, 5E, 5F) and an unstained lungtissue section (rows 3 and 4—FIGS. 5I, 5J, 5M, 5N), used as input to thedeep neural network. The third column (FIGS. 5C, 5G, 5K, 5O) shows thedigital/virtual staining results for these tissue samples. The lastcolumn (FIGS. 5D, 5H, 5L, 5P) shows the brightfield images of the sametissue sections, after the histochemical staining process.

FIG. 6A illustrates a graph of combined loss function vs. number ofiterations for random initialization and transfer learninginitialization. FIG. 6A illustrates how superior convergence is achievedusing transfer learning. A new deep neural network is initialized usingthe weights and biases learned from the salivary gland tissue sectionsto achieve virtual staining of thyroid tissue with H&E. Compared torandom initialization, transfer learning enables much fasterconvergence, also achieving a lower local minimum.

FIG. 6B illustrates network output images at different stages of thelearning process for both random initialization and transfer learning tobetter illustrate the impact of the transfer learning to translate thepresented approach to new tissue/stain combinations.

FIG. 6C illustrates the corresponding H&E chemically stained brightfieldimage.

FIG. 7A illustrates the virtual staining (H&E stain) of skin tissueusing the DAPI channel only.

FIG. 7B illustrates the virtual staining (H&E stain) of skin tissueusing the DAPI and Cy5 channels. Cy5 refers to a far-red-fluorescentlabel cyanine dye used to label biomolecules.

FIG. 7C illustrates the corresponding histologically stained (i.e.,chemically stained with H&E) tissue.

FIG. 8 illustrates the field-of-view matching and registration processof the auto-fluorescence images of unstained tissue samples with respectto the brightfield images of the same samples, after the chemicalstaining process.

FIG. 9 schematically illustrates the training process of the virtualstaining network using a GAN.

FIG. 10 illustrates the generative adversarial network (GAN)architecture for the generator and discriminator according to oneembodiment.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

FIG. 1 schematically illustrates one embodiment of a system 2 foroutputting digitally stained images 40 from an input microscope image 20of a sample 22. As explained herein, the input image 20 is afluorescence image 20 of a sample 22 (such as tissue in one embodiment)that is not stained or labeled with a fluorescent stain or label.Namely, the input image 20 is an autofluorescence image 20 of the sample22 in which the fluorescent light that is emitted by the sample 22 isthe result of one or more endogenous fluorophores or other endogenousemitters of frequency-shifted light contained therein. Frequency-shiftedlight is light that is emitted at a different frequency (or wavelength)that differs from the incident frequency (or wavelength). Endogenousfluorophores or endogenous emitters of frequency-shifted light mayinclude molecules, compounds, complexes, molecular species,biomolecules, pigments, tissues, and the like. In some embodiments, theinput image 20 (e.g., the raw fluorescent image) is subject to one ormore linear or non-linear pre-processing operations selected fromcontrast enhancement, contrast reversal, image filtering. The systemincludes a computing device 100 that contains one or more processors 102therein and image processing software 104 that incorporates the trained,deep neural network 10 (e.g., a convolutional neural network asexplained herein in one or more embodiments). The computing device 100may include, as explained herein, a personal computer, laptop, mobilecomputing device, remote server, or the like, although other computingdevices may be used (e.g., devices that incorporate one or more graphicprocessing units (GPUs)) or other application specific integratedcircuits (ASICs). GPUs or ASICs can be used to accelerate training aswell as final image output. The computing device 100 may be associatedwith or connected to a monitor or display 106 that is used to displaythe digitally stained images 40. The display 106 may be used to displaya Graphical User Interface (GUI) that is used by the user to display andview the digitally stained images 40. In one embodiment, the user may beable to trigger or toggle manually between multiple differentdigital/virtual stains for a particular sample 22 using, for example,the GUI. Alternatively, the triggering or toggling between differentstains may be done automatically by the computing device 100. In onepreferred embodiment, the trained, deep neural network 10 is aConvolution Neural Network (CNN).

For example, in one preferred embodiment as is described herein, thetrained, deep neural network 10 is trained using a GAN model. In aGAN-trained deep neural network 10, two models are used for training. Agenerative model is used that captures data distribution while a secondmodel estimates the probability that a sample came from the trainingdata rather than from the generative model. Details regarding GAN may befound in Goodfellow et al., Generative Adversarial Nets., Advances inNeural Information Processing Systems, 27, pp. 2672-2680 (2014), whichis incorporated by reference herein. Network training of the deep neuralnetwork 10 (e.g., GAN) may be performed the same or different computingdevice 100. For example, in one embodiment a personal computer may beused to train the GAN although such training may take a considerableamount of time. To accelerate this training process, one or morededicated GPUs may be used for training. As explained herein, suchtraining and testing was performed on GPUs obtained from a commerciallyavailable graphics card. Once the deep neural network 10 has beentrained, the deep neural network 10 may be used or executed on adifferent computing device 110 which may include one with lesscomputational resources used for the training process (although GPUs mayalso be integrated into execution of the trained deep neural network10).

The image processing software 104 can be implemented using Python andTensorFlow although other software packages and platforms may be used.The trained deep neural network 10 is not limited to a particularsoftware platform or programming language and the trained deep neuralnetwork 10 may be executed using any number of commercially availablesoftware languages or platforms. The image processing software 104 thatincorporates or runs in coordination with the trained, deep neuralnetwork 10 may be run in a local environment or a remove cloud-typeenvironment. In some embodiments, some functionality of the imageprocessing software 104 may run in one particular language or platform(e.g., image normalization) while the trained deep neural network 10 mayrun in another particular language or platform. Nonetheless, bothoperations are carried out by image processing software 104.

As seen in FIG. 1 , in one embodiment, the trained, deep neural network10 receives a single fluorescence image 20 of an unlabeled sample 22. Inother embodiments, for example, where multiple excitation channels areused (see melanin discussion herein), there may be multiple fluorescenceimages 20 of the unlabeled sample 22 that are input to the trained, deepneural network 10 (e.g., one image per channel). The fluorescence images20 may include a wide-field fluorescence image 20 of an unlabeled tissuesample 22. Wide-field is meant to indicate that a wide field-of-view(FOV) is obtained by scanning of a smaller FOV, with the wide FOV beingin the size range of 10-2,000 mm². For example, smaller FOVs may beobtained by a scanning fluorescent microscope 110 that uses imageprocessing software 104 to digitally stitch the smaller FOVs together tocreate a wider FOV. Wide FOVs, for example, can be used to obtain wholeslide images (WSI) of the sample 22. The fluorescence image is obtainedusing an imaging device 110. For the fluorescent embodiments describedherein, this may include a fluorescence microscope 110. The fluorescentmicroscope 110 includes an excitation light source that illuminates thesample 22 as well as one or more image sensor(s) (e.g., CMOS imagesensors) for capturing fluorescent light that is emitted by fluorophoresor other endogenous emitters of frequency-shifted light contained in thesample 22. The fluorescence microscope 110 may, in some embodiments,include the ability to illuminate the sample 22 with excitation light atmultiple different wavelengths or wavelength ranges/bands. This may beaccomplished using multiple different light sources and/or differentfilter sets (e.g., standard UV or near-UV excitation/emission filtersets). In addition, the fluorescence microscope 110 may include, in someembodiments, multiple filter sets that can filter different emissionbands. For example, in some embodiments, multiple fluorescence images 20may be captured, each captured at a different emission band using adifferent filter set.

The sample 22 may include, in some embodiments, a portion of tissue thatis disposed on or in a substrate 23. The substrate 23 may include anoptically transparent substrate in some embodiments (e.g., a glass orplastic slide or the like). The sample 22 may include a tissue sectionsthat are cut into thin sections using a microtome device or the like.Thin sections of tissue 22 can be considered a weakly scattering phaseobject, having limited amplitude contrast modulation under brightfieldillumination. The sample 22 may be imaged with or without a coverglass/cover slip. The sample may involve frozen sections or paraffin(wax) sections. The tissue sample 22 may be fixed (e.g., using formalin)or unfixed. The tissue sample 22 may include mammalian (e.g., human oranimal) tissue or plant tissue. The sample 22 may also include otherbiological samples, environmental samples, and the like. Examplesinclude particles, cells, cell organelles, pathogens, or othermicro-scale objects of interest (those with micrometer-sized dimensionsor smaller). The sample 22 may include smears of biological fluids ortissue. These include, for instance, blood smears, Papanicolaou or Papsmears. As explained herein, for the fluorescent-based embodiments, thesample 22 includes one or more naturally occurring or endogenousfluorophores that fluoresce and are captured by the fluorescentmicroscope device 110. Most plant and animal tissues show someautofluorescence when excited with ultraviolet or near ultra-violetlight. Endogenous fluorophores may include by way of illustrationproteins such as collagen, elastin, fatty acids, vitamins, flavins,porphyrins, lipofuscins, co-enzymes (e.g., NAD(P)H). In some optionalembodiments, exogenously added fluorescent labels or other exogenousemitters of light may also be added. As explained herein, the sample 22may also contain other endogenous emitters of frequency-shifted light.

The trained, deep neural network 10 in response to the input image 20outputs or generates a digitally stained or labelled output image 40.The digitally stained output image 40 has “staining” that has beendigitally integrated into the stained output image 40 using the trained,deep neural network 10. In some embodiments, such as those involvedtissue sections, the trained, deep neural network 10 appears to askilled observer (e.g., a trained histopathologist) to be substantiallyequivalent to a corresponding brightfield image of the same tissuesection sample 22 that has been chemically stained. Indeed, as explainedherein, the experimental results obtained using the trained, deep neuralnetwork 10 show that trained pathologists were able to recognizehistopathologic features with both staining techniques (chemicallystained vs. digitally/virtually stained) and with a high degree ofagreement between the techniques, without a clear preferable stainingtechnique (virtual vs. histological). This digital or virtual stainingof the tissue section sample 22 appears just like the tissue sectionsample 22 had undergone histochemical staining even though no suchstaining operation was conducted.

FIG. 2 schematically illustrates the operations involved in a typicalfluorescent-based embodiment. As seen in FIG. 2 , a sample 22 such as anunstained tissue section is obtained. This may be obtained from livingtissue such as through a biopsy B or the like. The unstained tissuesection sample 22 is then subject to fluorescent imaging using afluorescence microscope 110 and generates a fluorescence image 20. Thisfluorescence image 20 is then input to a trained, deep neural network 10that then promptly outputs a digitally stained image 40 of the tissuesection sample 22. This digitally stained image 40 closely resembles theappearance of a brightfield image of the same tissue section sample 22had the actual tissue section sample 22 be subject to histochemicalstaining. FIG. 2 illustrates (using dashed arrows) the conventionalprocess whereby the tissue section sample 22 is subject to histochemicalstaining 44 followed by conventional brightfield microscopic imaging 46to generate a conventional brightfield image 48 of the stained tissuesection sample 22. As seen in FIG. 2 , the digitally stained image 40closely resembles the actual chemically stained image 48. Similarresolution and color profiles are obtained using the digital stainingplatform described herein. This digitally stained image 40 may, asillustrated in FIG. 1 , be shown or displayed on a computer monitor 106but it should be appreciated the digitally stained image 40 may bedisplayed on any suitable display (e.g., computer monitor, tabletcomputer, mobile computing device, mobile phone, etc.). A GUI may bedisplayed on the computer monitor 106 so that the user may view andoptionally interact with the digitally stained image 40 (e.g., zoom,cut, highlight, mark, adjust exposure, and the like).

Experimental—Digital Staining of Label Free Tissue UsingAuto-Fluorescence

Virtual Staining of Tissue Samples

The system 2 and methods described herein was tested and demonstratedusing different combinations of tissue section samples 22 and stains.Following the training of a CNN-based deep neural network 10 itsinference was blindly tested by feeding it with the auto-fluorescenceimages 20 of label-free tissue sections 22 that did not overlap with theimages that were used in the training or validation sets. FIGS. 4A-4Hillustrates the results for a salivary gland tissue section, which wasdigitally/virtually stained to match H&E stained brightfield images 48(i.e., the ground truth images) of the same sample 22. These resultsdemonstrate the capability of the system 2 to transform a fluorescenceimage 20 of a label-free tissue section 22 into a brightfield equivalentimage 40, showing the correct color scheme that is expected from an H&Estained tissue, containing various constituents such as epithelioidcells, cell nuclei, nucleoli, stroma, and collagen. Evaluation of bothFIGS. 3C and 3D show the H&E stains demonstrate a small island ofinfiltrating tumor cells within subcutaneous fibro-adipose tissue. Notethe nuclear detail, including distinction of nucleoli (arrow) andchromatin texture, is clearly appreciated in both panels. Similarly, inFIGS. 3G and 3H, the H&E stains demonstrate infiltrating squamous cellcarcinoma. The desmoplastic reaction with edematous myxoid change(asterisk) in the adjacent stroma is clearly identifiable in bothstains.

Next, the deep network 10 was trained to digitally/virtually stain othertissue types with two different stains, i.e., the Jones methenaminesilver stain (kidney) and the Masson's Trichrome stain (liver and lung).FIGS. 4A-4H and 5A-5P summarize the results for deep learning-baseddigital/virtual staining of these tissue sections 22, which very wellmatch to the brightfield images 48 of the same samples 22, capturedafter the histochemical staining process. These results illustrate thatthe trained deep neural network 10 is capable of inferring the stainingpatterns of different types of histology stains used for differenttissue types, from a single fluorescence image 20 of a label-freespecimen (i.e., without any histochemical stains). With the same overallconclusion as in FIGS. 3A-3H, it was also confirmed by a pathologistthat the neural network output images FIGS. 4C and 5G correctly revealthe histological features corresponding to hepatocytes, sinusoidalspaces, collagen and fat droplets (FIG. 5G), consistent with the waythat they appear in the brightfield images 48 of the same tissue samples22, captured after the chemical staining (FIGS. 5D and 5H). Similarly,it was also confirmed by the same expert that the deep neural networkoutput images 40 reported in FIGS. 5K and 5O (lung) reveal consistentlystained histological features corresponding to vessels, collagen andalveolar spaces as they appear in the brightfield images 48 of the sametissue sample 22 imaged after the chemical staining (FIGS. 6L and 6P).

The digitally/virtually-stained output images 40 from the trained, deepneural network 10 were compared to the standard histochemical stainingimages 48 for diagnosing multiple types of conditions on multiple typesof tissues, which were either Formalin-Fixed Paraffin-Embedded (FFPE) orfrozen sections. The results are summarized in Table 1 below. Theanalysis of fifteen (15) tissue sections by four board certifiedpathologists (who were not aware of the virtual staining technique)demonstrated 100% non-major discordance, defined as no clinicallysignificant difference in diagnosis among professional observers. The“time to diagnosis” varied considerably among observers, from an averageof 10 seconds-per-image for observer 2 to 276 seconds-per-image forobserver 3. However, the intra-observer variability was very minor andtended towards shorter time to diagnosis with the virtually-stainedslide images 40 for all the observers except observer 2 which was equal,i.e., ˜10 seconds-per-image for both the virtual slide image 40 and thehistology stained slide image 48. These indicate very similar diagnosticutility between the two image modalities.

TABLE 1 Serial Tissue, num- fixation, type Patholo- Histochemically/Time to ber of stain gist # Virtually stained Diagnosis diagnose 1Ovary, Frozen 1 VS Adenocarcinoma 30 sec section, H&E 2 VS Borderlineserous tumor 15 sec 3 HS Mucinous adenocarcinoma 10 min 4 HSAdenocarcinoma, endometrioid 2 min 2 Ovary, Frozen 1 VS Benign ovary 10sec section, H&E 2 VS Benign ovary 10 sec 3 HS Normal ovary with corpusluteal cyst 15 min 4 HS Normal 1 min 3 Salivary 1 VS Benign salivaryglands with mild chronic 10 sec Gland, FFPE, inflammation H&E 2 VSBenign parotid tissue 5 sec 3 HS Normal salivary gland 1 min 4 HS Nohistopathologic abnormality 1 min 4 Salivary 1 HS Pleomorphic adenoma 5sec Gland, Frozen 2 HS Pleomorphic adenoma 5 sec section, H&E 3 VSPleomorphic adenoma 3 min 4 VS Pleomorphic adenoma 2 sec 5 Salivary 1 HSMucoepidermoid carcinoma, low grade 5 sec Gland, FFPE, 2 HS Salivaryduct carcinoma 5 sec H&E 3 VS Mucoepidermoid carcinoma 10 min 4 VSMucoepidermoid Carcinoma 10 sec 6 Breast, FFPE, 1 VS Invasive ductalcarcinoma and DCIS 15 sec H&E 2 VS Ductal carcinoma 10 sec 3 HS Invasiveductal carcinoma with DCIS 2 min 4 HS Invasive carcinoma 1 minute 7Skin, FFPE, 1 HS Malignant melanoma 30 sec H&E 2 HS melanoma 30 sec 3 VSMelanoma 5 min 4 VS Melanoma 1 min 8 Prostate, 1 HS Prostaticadenocarcinoma 3 + 4 1 min FFPE, H&E 2 HS Prostatic adenocarcinoma 4 + 35 sec 3 VS Prostatic adenocarcinoma, Gleason pattern 5 min 3 + 4 4 VSHG-PIN with cribiforming vs carcinoma 5 min 9 Liver, FFPE, 1 VS Benignliver with mild steatosis 10 sec Masson's 2 VS Benign liver withsteatosis 5 sec trichrome 3 HS Hepatosteatosis, predominantly 3 minmacrovesicular 4 HS Minimal steatosis, no fibrosis 5 min 10 Liver, FFPE,1 HS Benign liver with bridging fibrosis 10 sec Masson's 2 HS Benignliver, bridging fibrosis 5 sec trichrome 3 VS Moderate cirrhosis 1 min 4VS Mild portal inflammation, focal bridging 5 minutes fibrosis (Stage2-3) 11 Salivary 1 VS Carcinoma 5 sec Gland, FFPE, 2 VS Intraductal ca20 sec H&E 3 HS Poorly differentiated carcinoma 1 min 4 HS Low-gradesalivary gland neoplasm 1 minute 12 Salivary 1 HS Adenocarcinoma 5 secGland, FFPE, 2 HS Salivary duct carcinoma 5 sec H&E 3 VS Salivary ductcarcinoma 2 min 4 VS Low-grade salivary gland neoplasm 1 minute 13Thyroid, 1 VS Papillary thyroid carcinoma, tall cell type 10 sec FFPE,H&E 2 VS Papillary thyroid ca, tall cell 20 sec 3 HS Papillary thyroidcarcinoma, tall cell 5 min variant 4 HS PTC 10 sec 14 Thyroid, 1 HSPapillary thyroid carcinoma 5 sec FFPE, H&E 2 HS Medullary ca 5 sec 3 VSPapillary thyroid carcinoma, oncocytic 7 min variant 4 VS PTC 10 sec 15Thyroid, 1 VS Papillary thyroid carcinoma 5 sec FFPE, H&E 2 VS Papillarythyroid ca 5 sec 3 HS Papillary thyroid carcinoma 1 min 4 HS PTC 10 sec

Blind Evaluation of Staining Efficacy for Whole Slide Images (WSIs)

After evaluating the differences in tissue section and stains, theability of the virtual staining system 2 was tested in the specializedstaining histology workflow. In particular, the autofluorescencedistribution of 15 label-free samples of liver tissue sections and 13label-free tissue sections of kidney were imaged with a 20×/0.75NAobjective lens. All liver and kidney tissue sections were obtained fromdifferent patients and included both small biopsies and largerresections. All the tissue sections were obtained from FFPE but notcover slipped. After the autofluorescence scanning, the tissue sectionswere histologically stained with Masson's Trichrome (4 μm liver tissuesections) and Jones' stain (2 μm kidney tissue sections). The WSIs werethen divided into training and test sets. For the liver slides cohort, 7WSIs were used for training the virtual staining algorithm and 8 WSIswere used for blind testing; for the kidney slides cohort, 6 WSIs wereused for training the algorithm and 7 WSIs were used for testing. Thestudy pathologists were blinded to the staining techniques for each WSIand were asked to apply a 1-4 number grade for the quality of thedifferent stains: 4=perfect, 3=very good, 2=acceptable, 1=unacceptable.Secondly, the study pathologists applied the same score scale (1-4) forspecific features: nuclear detail (ND), cytoplasmic detail (CD) andextracellular fibrosis (EF), for liver only. These results aresummarized in Table 2 (Liver) and Table 3 (Kidney) below (winner isbolded). The data indicates that the pathologists were able to recognizehistopathologic features with both staining techniques and with a highdegree of agreement between the techniques, without a clear preferablestaining technique (virtual vs. histological).

TABLE 2 Tis. Pathologist 1 Pathologist 2 Pathologist 3 Average # ND CDEF SQ ND CD EF SQ ND CD EF SQ ND CD EF SQ 1-HS 3 2 1 1 4 4 3 4 1 1 1 32.67 2.33 1.67 2.67 1-VS 3 3 3 3 3 3 2 3 2 2 3 3 2.67 2.67 2.67 3.002-HS 3 2 4 4 4 4 3 4 1 2 2 2 2.67 2.67 3.00 3.33 2-VS 3 3 4 4 4 3 3 3 22 3 3 3.00 2.67 3.33 3.33 3-HS 3 3 2 2 3 3 4 3 1 1 1 1 2.33 2.33 2.332.00 3-VS 3 2 1 1 3 3 1 4 1 1 1 1 2.33 2.00 1.00 2.00 4-HS 3 2 4 4 3 4 44 1 2 1 2 2.33 2.67 3.00 3.33 4-VS 3 3 4 4 4 3 4 4 2 2 3 3 3.00 2.673.67 3.67 5-HS 3 3 4 4 3 3 2 1 1 3 2 2 2.33 3.00 2.67 2.33 5-VS 3 2 3 33 3 4 2 2 1 3 3 2.67 2.00 3.33 2.67 6-HS 3 2 3 3 4 4 4 3 2 2 2 2 3.002.67 3.00 2.67 6-VS 3 3 4 3 4 3 4 3 1 1 1 1 2.67 2.33 3.00 2.33 7-HS 3 34 4 3 4 4 3 2 1 2 2 2.67 2.67 3.33 3.00 7-VS 3 2 3 3 4 4 4 3 2 2 3 33.00 2.67 3.33 3.00 8-HS 3 3 4 4 4 4 4 3 1 1 1 1 2.67 2.67 3.00 2.678-VS 3 2 4 4 4 3 4 4 2 2 3 2 3.00 2.33 3.67 3.33

TABLE 3 Tissue Pathologist 1 Pathologist 2 Pathologist 3 Average # ND CDSQ ND CD SQ ND CD SQ ND CD SQ 1-HS 3 3 3 2 2 4 2 2 2 2.33 2.33 3.00 1-VS2 3 3 3 3 4 3 3 3 2.67 3.00 3.33 2-HS 2 4 4 3 3 2 1 1 2 2.00 2.67 2.672-VS 2 3 4 3 3 3 1 2 3 2.00 2.67 3.33 3-HS 2 3 3 3 3 2 2 3 4 2.33 3.003.00 3-VS 2 3 3 3 3 3 1 2 3 2.00 2.67 3.00 4-HS 3 3 3 2 2 2 1 2 3 2.002.33 2.67 4-VS 3 3 3 2 2 3 1 2 2 2.00 2.33 2.67 5-HS 3 3 2 3 3 1 3 3 33.00 3.00 2.00 5-VS 3 3 2 4 3 4 3 3 4 3.33 3.00 3.33 6-HS 2 3 3 3 3 1 22 2 2.33 2.67 2.00 6-VS 2 2 3 2 2 2 2 2 2 2.00 2.00 2.33 7-HS 3 3 2 3 22 3 3 3 3.00 2.67 2.33 7-VS 3 3 2 4 3 1 3 2 3 3.33 2.67 2.00

Quantification of the Network Output Image Quality

Next, beyond the visual comparison provided in FIGS. 3A-3H, 4A-4H,5A-5P, the results of the trained deep neural network 10 were quantifiedby first calculating the pixel-level differences between the brightfieldimages 48 of the chemically stained samples 22 and thedigitally/virtually stained images 40 that are synthesized using thedeep neural network 10 without the use of any labels/stains. Table 4below summarizes this comparison for different combinations of tissuetypes and stains, using the YCbCr color space, where the chromacomponents Cb and Cr entirely define the color, and Y defines thebrightness component of the image. The results of this comparison revealthat the average difference between these two sets of images is <˜5% and<˜16%, for the chroma (Cb, Cr) and brightness (Y) channels,respectively. Next, a second metric was used to further quantify thecomparison, i.e., the structural similarity index (SSIM), which is ingeneral used to predict the score that a human observer will give for animage, in comparison to a reference image (Equation 8 herein). SSIMranges between 0 and 1, where 1 defines the score for identical images.The results of this SSIM quantification are also summarized in Table 4,which very well illustrates the strong structural similarity between thenetwork output images 40 and the brightfield images 48 of the chemicallystained samples.

TABLE 4 Virtual histological staining Y Cb Cr using Number differencedifference difference a deep of test SSIM (%) (%) (%) network imagesmean std mean std mean std mean std Salivary 10 0.826 0.059 11.5 9.0 2.52.4 2.5 2.5 gland (H&E) Thyroid 30 0.789 0.044 10.1 7.9 3.4 2.7 2.8 2.7(H&E) Thyroid 30 0.839 0.029 14.0 8.4 2.4 2.2 2.6 2.6 (H&E, transferlearning) Liver 30 0.847 0.023 11.0 8.9 3.1 2.7 4.0 3.5 (Masson’sTriehrome) Lung 48 0.776 0.039 15.9 11.7 4.0 3.6 5.3 4.9 (Masson’sTriehrome) Kidney 30 0.841 0.021 16.1 10.4 2.5 2.2 3.6 3.4 (Jones Stain)

One should note that the brightfield images 48 of the chemically stainedtissue samples 22 in fact do not provide the true gold standard for thisspecific SSIM and YCbCr analysis of the network output images 40,because there are uncontrolled variations and structural changes thatthe tissue undergoes during the histochemical staining process andrelated dehydration and clearing steps. Another variation that wasnoticed for some of the images was that the automated microscopescanning software selected different auto-focusing planes for the twoimaging modalities. All these variations create some challenges for theabsolute quantitative comparison of the two sets of images (i.e., thenetwork output 40 for a label-free tissue vs. the brightfield image 48of the same tissue after the histological staining process).

Staining Standardization

An interesting by-product of the digital/virtual staining system 2 canbe staining standardization. In other words, the trained deep neuralnetwork 10 converges to a “common stain” colorization scheme whereby thevariation in the histologically stained tissue images 48 is higher thanthat of the virtually stained tissue images 40. The colorization of thevirtual stain is solely the result of its training (i.e., the goldstandard histological staining used during the training phase) and canbe further adjusted based on the preferences of pathologists, byretraining the network with a new stain colorization. Such “improved”training can be created from scratch or accelerated through transferlearning. This potential staining standardization using deep learningcan remedy the negative effects of human-to-human variations atdifferent stages of the sample preparation, create a common ground amongdifferent clinical laboratories, enhance the diagnostic workflow forclinicians as well as assist the development of new algorithms such asautomatic tissue metastasis detection or grading of different types ofcancer, among others.

Transfer Learning to Other Tissue-Stain Combinations

Using the concept of transfer learning, the training procedure for newtissue and/or stain types can converge much faster, while also reachingan improved performance, i.e., a better local minimum in the trainingcost/loss function. This means, a pre-learnt CNN model deep neuralnetwork 10, from a different tissue-stain combination, can be used toinitialize the deep neural network 10 to statistically learn virtualstaining of a new combination. FIGS. 6A-6C shows the favorableattributes of such an approach: a new deep neural network 10 was trainedto virtually stain the auto-fluorescence images 20 of unstained thyroidtissue sections, and it was initialized using the weights and biases ofanother deep neural network 10 that was previously trained for H&Evirtual staining of the salivary gland. The evolution of the loss metricas a function of the number of iterations used in the training phaseclearly demonstrates that the new thyroid deep network 10 rapidlyconverges to a lower minimum in comparison to the same networkarchitecture which was trained from scratch, using random initializationas seen in FIG. 6A. FIG. 6B compares the output images 40 of thisthyroid network 10 at different stages of its learning process, whichfurther illustrates the impact of transfer learning to rapidly adapt thepresented approach to new tissue/stain combinations. The network outputimages 40, after the training phase with e.g., >6,000 iterations, revealthat cell nuclei show irregular contours, nuclear grooves, and chromatinpallor, suggestive of papillary thyroid carcinoma; cells also show mildto moderate amounts of eosinophilic granular cytoplasm and thefibrovascular core at the network output image shows increasedinflammatory cells including lymphocytes and plasma cells. FIG. 6Cillustrates the corresponding H&E chemically stained brightfield image48.

Using Multiple Fluorescent Channels at Different Resolutions

The method of using the trained, deep neural network 10 can be combinedwith other excitation wavelengths and/or imaging modalities in order toenhance its inference performance for different tissue constituents. Forexample, melanin detection on a skin tissue section sample using virtualH&E staining was tried. However, melanin was not clearly identified inthe output of the network, as it presents a weak auto-fluorescent signalat DAPI excitation/emission wavelengths measured in the experimentalsystem described herein. One potential method to increase theautofluorescence of melanin is to image the samples while they are in anoxidizing solution. However, a more practical alternative was used wherean additional autofluorescence channel was employed, originating frome.g., Cy5 filter (excitation 628 nm/emission 692 nm) such that themelanin signal can be enhanced and accurately inferred in the trained,deep neural network 10. By training the network 10 using both the DAPIand Cy5 autofluorescence channels, the trained, deep neural network 10was able to successfully determine where melanin occurs in the sample,as illustrated in FIGS. 7A-7C. In contrast, when only the DAPI channelwas used (FIG. 7A), the network 10 was unable to determine the areasthat contain melanin (the areas appear white). Stated differently, theadditional autofluorescence information from the Cy5 channel was used bythe network 10 to distinguish melanin from the background tissue. Forthe results that are shown in FIGS. 7A-7C, the images 20 were acquiredusing a lower resolution objective lens (10×/0.45NA) for the Cy5channel, to supplement the high-resolution DAPI scan (20×/0.75NA), as itwas hypothesized that most necessary information is found in thehigh-resolution DAPI scan and the additional information (for example,the melanin presence) can be encoded with the lower resolution scan. Inthis manner, two different channels were used with one of the channelsbeing used at a lower resolution to identify the melanin. This mayrequire multiple scanning passes of the sample 22 with the fluorescentmicroscope 110. In yet another multi-channel embodiment, multiple images20 may be fed to the trained, deep neural network 10. This may include,for example, raw fluorescent images in combination with one or moreimages that have undergone linear or non-linear image pre-processingsuch as contrast enhancement, contrast reversal, and image filtering.

The system 2 and methods described herein show the ability todigitally/virtually stain label-free tissue sections 22, using asupervised deep learning technique that uses a single fluorescence image20 of the sample as input, captured by a standard fluorescencemicroscope 110 and filter set (in other embodiments multiplefluorescence images 20 are input when multiple fluorescence channels areused). This statistical learning-based method has the potential torestructure the clinical workflow in histopathology and can benefit fromvarious imaging modalities such as fluorescence microscopy, non-linearmicroscopy, holographic microscopy, stimulated Raman scatteringmicroscopy, and optical coherence tomography, among others, topotentially provide a digital alternative to the standard practice ofhistochemical staining of tissue samples 22. Here, the method wasdemonstrated using fixed unstained tissue samples 22 to provide ameaningful comparison to chemically stained tissue samples, which isessential to train the deep neural network 10 as well as to blindly testthe performance of the network output against the clinically-approvedmethod. However, the presented deep learning-based approach is broadlyapplicable to different types and states of a sample 22 includingun-sectioned, fresh tissue samples (e.g., following a biopsy procedure)without the use of any labels or stains. Following its training, thedeep neural network 10 can be used to digitally/virtually stain theimages of label-free fresh tissue samples 22, acquired using e.g., UV ordeep UV excitation or even nonlinear microscopy modalities. For example,Raman microscopy can provide very rich label-free biochemical signaturesthat can further enhance the effectiveness of the virtual staining thatthe neural network learns.

An important part of the training process involves matching thefluorescence images 20 of label-free tissue samples 22 and theircorresponding brightfield images 48 after the histochemical stainingprocess (i.e., chemically stained images). One should note that duringthe staining process and related steps, some tissue constitutes can belost or deformed in a way that will mislead the loss/cost function inthe training phase. This, however, is only a training and validationrelated challenge and does not pose any limitations on the practice of awell-trained deep neural network 10 for virtual staining of label-freetissue samples 22. To ensure the quality of the training and validationphase and minimize the impact of this challenge on the network'sperformance, a threshold was established for an acceptable correlationvalue between the two sets of images (i.e., before and after thehistochemical staining process) and eliminated the non-matching imagepairs from the training/validation set to make sure that the deep neuralnetwork 10 learns the real signal, not the perturbations to the tissuemorphology due to the chemical staining process. In fact, this processof cleaning the training/validation image data can be done iteratively:one can start with a rough elimination of the obviously altered samplesand accordingly converge on a neural network 10 that is trained. Afterthis initial training phase, the output images 40 of each sample in theavailable image set can be screened against their correspondingbrightfield images 48 to set a more refined threshold to reject someadditional images and further clean the training/validation image set.With a few iterations of this process, one can, not only further refinethe image set, but also improve the performance of the final traineddeep neural network 10.

The methodology described above will mitigate some of the trainingchallenges due to random loss of some tissue features after thehistological staining process. In fact, this highlights anothermotivation to skip the laborious and costly procedures that are involvedin histochemical staining as it will be easier to preserve the localtissue histology in a label-free method, without the need for an expertto handle some of the delicate procedures of the staining process, whichsometimes also requires observing the tissue under a microscope.

Using a PC desktop, the training phase of the deep neural network 10takes a considerable amount of time (e.g., ˜13 hours for the salivarygland network). However, this entire process can be significantlyaccelerated by using dedicated computer hardware, based on GPUs.Furthermore, as already emphasized in FIGS. 6A-6C, transfer learningprovides a “warm start” to the training phase of a new tissue/staincombination, making the entire process significantly faster. Once thedeep neural network 10 has been trained, the digital/virtual staining ofa sample image 40 is performed in a single, non-iterative manner, whichdoes not require a trial-and-error approach or any parameter tuning toachieve the optimal result. Based on its feed-forward and non-iterativearchitecture, the deep neural network 10 rapidly outputs a virtuallystained image in less than one second (e.g., 0.59 sec, corresponding toa sample field-of-view of ˜0.33 mm×0.33 mm). With further GPU-basedacceleration, it has the potential to achieve real-time or nearreal-time performance in outputting digitally/virtually stained images40 which might especially be useful in the operating room or for in vivoimaging applications.

The digital/virtual staining procedure that is implemented is based ontraining a separate CNN deep neural network 10 for each tissue/staincombination. If one feeds a CNN-based deep neural network 10 with theauto-fluorescence images 20 having different tissue/stain combinations,it will not perform as desired. This, however, is not a limitationbecause for histology applications, the tissue type and stain type arepre-determined for each sample 22 of interest, and therefore, a specificCNN selection for creating the digitally/virtually stained image 40 froman auto-fluorescence image 20 of the unlabeled sample 22 does notrequire an additional information or resource. Of course, a more generalCNN model can be learnt for multiple tissue/stain combinations by e.g.,increasing the number of trained parameters in the model, at the cost ofa possible increase in the training and inference times. Another avenueis the potential of the system 2 and method to perform multiple virtualstains on the same unlabeled tissue type.

A significant advantage of the system 2 is that it is quite flexible. Itcan accommodate feedback to statistically mend its performance if adiagnostic failure is detected through a clinical comparison, byaccordingly penalizing such failures as they are caught. This iterativetraining and transfer learning cycle, based on clinical evaluations ofthe performance of the network output, will help optimize the robustnessand clinical impact of the presented approach. Finally, this method andsystem 2 may be used for micro-guiding molecular analysis at theunstained tissue level, by locally identifying regions of interest basedon virtual staining, and using this information to guide subsequentanalysis of the tissue for e.g., micro-immunohistochemistry orsequencing. This type of virtual micro-guidance on an unlabeled tissuesample can facilitate high-throughput identification of sub-types ofdiseases, also helping the development of customized therapies forpatients.

Sample Preparation

Formalin-fixed paraffin-embedded 2 μm thick tissue sections weredeparaffinized using Xylene and mounted on a standard glass slide usingCytoseal™ (Thermo-Fisher Scientific, Waltham, MA USA), followed byplacing a coverslip (Fisherfinest™, 24×50-1, Fisher Scientific,Pittsburgh, PA USA). Following the initial auto-fluorescence imagingprocess (using a DAPI excitation and emission filter set) of theunlabeled tissue sample, the slide was then put into Xylene forapproximately 48 hours or until the coverslip can be removed withoutdamaging the tissue. Once the coverslip is removed the slide was dipped(approximately 30 dips) in absolute alcohol, 95% alcohol and then washedin D.I. water for ˜1 min. This step was followed by the correspondingstaining procedures, used for H&E, Masson's Trichrome or Jones stains.This tissue processing path is only used for the training and validationof the approach and is not needed after the network has been trained. Totest the system and method, different tissue and stain combinations wereused: the salivary gland and thyroid tissue sections were stained withH&E, kidney tissue sections were stained with Jones stain, while theliver and lung tissue sections were stained with Masson's trichrome.

In the WSI study, the FFPE 2-4 μm thick tissue sections were not coverslipped during the autofluorescence imaging stage. Following theautofluorescence imaging, the tissue samples were histologically stainedas described above (Masson's Trichrome for the liver and Jones for thekidney tissue sections). The unstained frozen samples were prepared byembedding the tissue section in O.C.T. (Tissue Tek, SAKURA FINETEK USAINC) and dipped in 2-Methylbutane with dry ice. The frozen section wasthen cut to 4 μm sections and was put in a freezer until it was imaged.Following the imaging process, the tissue section was washed with 70%alcohol, H&E stained and cover slipped. The samples were obtained fromthe Translational Pathology Core Laboratory (TPCL) and were prepared bythe Histology Lab at UCLA. The kidney tissue sections of diabetic andnon-diabetic patients were obtained under IRB 18-001029 (UCLA). All thesamples were obtained after de-identification of the patient relatedinformation, and were prepared from existing specimen. Therefore, thiswork did not interfere with standard practices of care or samplecollection procedures.

Data Acquisition

The label-free tissue auto-fluorescence images 20 were captured using aconventional fluorescence microscope 110 (1X83, Olympus Corporation,Tokyo, Japan) equipped with a motorized stage, where the imageacquisition process was controlled by MetaMorph® microscope automationsoftware (Molecular Devices, LLC). The unstained tissue samples wereexcited with near UV light and imaged using a DAPI filter cube(OSFI3-DAPI-5060C, excitation wavelength 377 nm/50 nm bandwidth,emission wavelength 447 nm/60 nm bandwidth) with a 40×/0.95NA objectivelens (Olympus UPLSAPO 40×2/0.95NA, WD0.18) or 20×/0.75NA objective lens(Olympus UPLSAPO 20×/0.75NA, WD0.65). For the melanin inference, theautofluorescence images of the samples were additionally acquired usinga Cy5 filter cube (CY5-4040C-OFX, excitation wavelength 628 nm/40 nmbandwidth, emission wavelength 692 nm/40 nm bandwidth) with a 10×/0.4NAobjective lens (Olympus UPLSAPO10X2). Each auto-fluorescence image wascaptured with a scientific CMOS sensor (ORCA-flash4.0 v2, HamamatsuPhotonics K.K., Shizuoka Prefecture, Japan) with an exposure time of˜500 ms. The brightfield images 48 (used for the training andvalidation) were acquired using a slide scanner microscope (Aperio AT,Leica Biosystems) using a 20×/0.75NA objective (Plan Apo), equipped witha 2× magnification adapter.

Image Pre-Processing and Alignment

Since the deep neural network 10 aims to learn a statisticaltransformation between an auto-fluorescence image 20 of a chemicallyunstained tissue sample 22 and a brightfield image 48 of the same tissuesample 22 after the histochemical staining, it is important toaccurately match the FOV of the input and target images (i.e., unstainedauto-fluorescence image 20 and the stained bright-filed image 48). Anoverall scheme describing the global and local image registrationprocess is described in FIG. 8 which was implemented in MATLAB (TheMathWorks Inc., Natick, MA, USA). The first step in this process is tofind candidate features for matching unstained auto-fluorescence imagesand chemically stained brightfield images. For this, eachauto-fluorescence image 20 (2048×2048 pixels) is down-sampled to matchthe effective pixel size of the brightfield microscope images. Thisresults in a 1351×1351-pixel unstained auto-fluorescent tissue image,which is contrast enhanced by saturating the bottom 1% and the top 1% ofall the pixel values, and contrast reversed (image 20 a in FIG. 8 ) tobetter represent the color map of the grayscale converted whole slideimage. Then, a correlation patch process 60 is performed in which anormalized correlation score matrix is calculated by correlating eachone of the 1351×1351-pixel patches with the corresponding patch of thesame size, extracted from the whole slide gray-scale image 48 a. Theentry in this matrix with the highest score represents the most likelymatched FOV between the two imaging modalities. Using this information(which defines a pair of coordinates), the matched FOV of the originalwhole slide brightfield image 48 is cropped 48 c to create target images48 d. Following this FOV matching procedure 60, the auto-fluorescence 20and brightfield microscope images 48 are coarsely matched. However, theyare still not accurately registered at the individual pixel-level, dueto the slight mismatch in the sample placement at the two differentmicroscopic imaging experiments (auto-fluorescence, followed bybrightfield), which randomly causes a slight rotation angle (e.g., ˜1-2degrees) between the input and target images of the same sample.

The second part of the input-target matching process involves a globalregistration step 64, which corrects for this slight rotation anglebetween the auto-fluorescence and brightfield images. This is done byextracting feature vectors (descriptors) and their correspondinglocations from the image pairs, and matching the features by using theextracted descriptors. Then, a transformation matrix corresponding tothe matched pairs is found using the M-estimator Sample Consensus (MSAC)algorithm, which is a variant of the Random Sample Consensus (RANSAC)algorithm. Finally, the angle-corrected image 48 e is obtained byapplying this transformation matrix to the original brightfieldmicroscope image patch 48 d. Following the application of this rotation,the images 20 b, 48 e are further cropped by 100 pixels (50 pixels oneach side) to accommodate for undefined pixel values at the imageborders, due to the rotation angle correction.

Finally, for the local feature registration operation 68, an elasticimage registration, which matches the local features of both sets ofimages (auto-fluorescence 20 b vs. brightfield 48 e), by hierarchicallymatching the corresponding blocks, from large to small. A neural network71 is used to learn the transformation between the roughly matchedimages. This network 71 uses the same structure as the network 10 inFIG. 10 . A low number of iterations is used so that the network 71 onlylearns the accurate color mapping, and not any spatial transformationsbetween the input and label images. The calculated transformation mapfrom this step is finally applied to each brightfield image patch 48 e.At the end of these registration steps 60, 64, 68, the auto-fluorescenceimage patches 20 b and their corresponding brightfield tissue imagepatches 48 f are accurately matched to each other and can be used asinput and label pairs for the training of the deep neural network 10,allowing the network to solely focus on and learn the problem of virtualhistological staining.

For the 20× objective lens images (that were used for generating Table 2and Table 3 data) a similar process was used. Instead of down-samplingthe auto-fluorescence images 20, the bright-field microscope images 48were down-sampled to 75.85% of their original size so that they matchwith the lower magnification images. Furthermore, to create whole slideimages using these 20× images, additional shading correction andnormalization techniques were applied. Before being fed into the network71, each field-of-view was normalized by subtracting the mean valueacross the entire slide and dividing it by the standard deviationbetween pixel values. This normalizes the network input both within eachslide as well as between slides. Finally, shading correction was appliedto each image to account for the lower relative intensity measured atthe edges of each field-of-view.

Deep Neural Network Architecture and Training

In this work, a GAN architecture was used to learn the transformationfrom a label-free unstained auto-fluorescence input image 20 to thecorresponding brightfield image 48 of the chemically stained sample. Astandard convolutional neural network-based training learns to minimizea loss/cost function between the network's output and the target label.Thus, the choice of this loss function 69 (FIGS. 9 and 10 ) is acritical component of the deep network design. For instance, simplychoosing an l₂-norm penalty as a cost function will tend to generateblurry results, as the network averages a weighted probability of allthe plausible results; therefore, additional regularization terms aregenerally needed to guide the network to preserve the desired sharpsample features at the network's output. GANs avoid this problem bylearning a criterion that aims to accurately classify if the deepnetwork's output image is real or fake (i.e., correct in its virtualstaining or wrong). This makes the output images that are inconsistentwith the desired labels not to be tolerated, which makes the lossfunction to be adaptive to the data and the desired task at hand. Toachieve this goal, the GAN training procedure involves training of twodifferent networks, as shown in FIGS. 9 and 10 : (i) a generator network70, which in this case aims to learn the statistical transformationbetween the unstained auto-fluorescence input images 20 and thecorresponding brightfield images 48 of the same samples 12, after thehistological staining process; and (ii) a discriminator network 74 thatlearns how to discriminate between a true brightfield image of a stainedtissue section and the generator network's output image. Ultimately, thedesired result of this training process is a trained deep neural network10, which transforms an unstained auto-fluorescence input image 20 intoa digitally stained image 40 which will be indistinguishable from thestained brightfield image 48 of the same sample 22. For this task, theloss functions 69 of the generator 70 and discriminator 74 were definedas such:

_(generator) =MSE{z _(label) ,z _(output) }+λ×TV{z _(output)}+α×(1−D(z_(output)))²

_(discriminator) =D(z _(output))²+(1−D(z _(label)))²  (1)

where D refers to the discriminator network output, z_(label) denotesthe brightfield image of the chemically stained tissue, z_(output)denotes the output of the generator network. The generator loss functionbalances the pixel-wise mean squared error (MSE) of the generatornetwork output image with respect to its label, the total variation (TV)operator of the output image, and the discriminator network predictionof the output image, using the regularization parameters (λ, α) that areempirically set to different values, which accommodate for ˜2% and ˜20%of the pixel-wise MSE loss and the combined generator loss(l_(generator)), respectively. The TV operator of an image z is definedas:

$\begin{matrix}{{T{V(z)}} = {\sum\limits_{p}{\sum\limits_{q}\sqrt{\left( {z_{{p + l},q} - z_{p,q}} \right)^{2} + \left( {z_{p,{q + l}} - z_{p,q}} \right)^{2}}}}} & (2)\end{matrix}$

where p, q are pixel indices. Based on Eq. (1), the discriminatorattempts to minimize the output loss, while maximizing the probabilityof correctly classifying the real label (i.e., the brightfield image ofthe chemically stained tissue). Ideally, the discriminator network wouldaim to achieve D(z_(label))=1 and D(z_(output))=0, but if the generatoris successfully trained by the GAN, D(z_(output)) will ideally convergeto 0.5.

The generator deep neural network architecture 70 is detailed in FIG. 10. An input image 20 is processed by the network 70 in a multi-scalefashion, using down-sampling and up-sampling paths, helping the networkto learn the virtual staining task at various different scales. Thedown-sampling path consists of four individual steps (four blocks #1,#2, #3, #4), with each step containing one residual block, each of whichmaps a feature map x_(k) into feature map x_(k+1):x _(k+1) =x _(k)+LReLU[CONV_(k3){LReLU[CONV_(k2){LReLU[CONV_(k1) {x_(k)}]}]}]  (3)

where CONV{.} is the convolution operator (which includes the biasterms), k1, k2, and k3 denote the serial number of the convolutionlayers, and LReLU[.] is the non-linear activation function (i.e., aLeaky Rectified Linear Unit) that was used throughout the entirenetwork, defined as:

$\begin{matrix}{{{LReLU}(x)} = \left\{ \begin{matrix}x & {{{for}\mspace{14mu} x} > 0} \\{0.1x} & {otherwise}\end{matrix} \right.} & (4)\end{matrix}$

The number of the input channels for each level in the down-samplingpath was set to: 1, 64, 128, 256, while the number of the outputchannels in the down-sampling path was set to: 64, 128, 256, 512. Toavoid the dimension mismatch for each block, the feature map x_(k) waszero-padded to match the number of the channels in x_(k+1) Theconnection between each down-sampling level is a 2×2 average poolinglayer with a stride of 2 pixels that down-samples the feature maps by afactor of 4 (2-fold for in each direction). Following the output of thefourth down-sampling block, another convolutional layer (CL) maintainsthe number of the feature maps as 512, before connecting it to theup-sampling path. The up-sampling path consists of four, symmetric,up-sampling steps (#1, #2, #3, #4), with each step containing oneconvolutional block. The convolutional block operation, which mapsfeature map y_(k) into feature map y_(k+1), is given by:y _(k+1)=LReLU[CONV_(k6){LReLU[CONV_(k5){LReLU[CONV_(k4){CONACT(x _(k+1),US{y _(k)})}]}]}]  (5)

where CONCAT(.) is the concatenation between two feature maps whichmerges the number of channels, US{.} is the up-sampling operator, andk4, k5, and k6, denote the serial number of the convolution layers. Thenumber of the input channels for each level in the up-sampling path wasset to 1024, 512, 256, 128 and the number of the output channels foreach level in the up-sampling path was set to 256, 128, 64, 32,respectively. The last layer is a convolutional layer (CL) mapping 32channels into 3 channels, represented by the YCbCr color map. Both thegenerator and the discriminator networks were trained with a patch sizeof 256×256 pixels.

The discriminator network, summarized in FIG. 10 , receives three (3)input channels, corresponding to the YCbCr color space of an input image40YCbCr, 48YCbCr. This input is then transformed into a 64-channelrepresentation using a convolutional layer, which is followed by 5blocks of the following operator:z _(k+1)=LReLU[CONV_(k2){LReLU[CONV_(k1) {z _(k)}]}]  (6)

where k1, k2, denote the serial number of the convolutional layer. Thenumber of channels for each layer was 3, 64, 64, 128, 128, 256, 256,512, 512, 1024, 1024, 2048. The next layer was an average pooling layerwith a filter size that is equal to the patch size (256×256), whichresults in a vector with 2048 entries. The output of this averagepooling layer is then fed into two fully connected layers (FC) with thefollowing structure:z _(k+1) =FC[LReLU[FC{z _(k)}]]  (7)

where FC represents the fully connected layer, with learnable weightsand biases. The first fully connected layer outputs a vector with 2048entries, while the second one outputs a scalar value. This scalar valueis used as an input to a sigmoid activation function D(z)=1/(1+exp(−z))which calculates the probability (between 0 and 1) of the discriminatornetwork input to be real/genuine or fake, i.e., ideally D(z_(label))=1as illustrated by output 67 in FIG. 10 .

The convolution kernels throughout the GAN were set to be 3×3. Thesekernels were randomly initialized by using a truncated normaldistribution with a standard deviation of 0.05 and a mean of 0; all thenetwork biases were initialized as 0. The learnable parameters areupdated through the training stage of the deep neural network 10 by backpropagation (illustrated in dashed arrows of FIG. 10 ) using an adaptivemoment estimation (Adam) optimizer with learning rate 1×10⁻⁴ for thegenerator network 70 and 1×10⁻⁵ for the discriminator network 74. Also,for each iteration of the discriminator 74, there were 4 iterations ofthe generator network 70, to avoid training stagnation following apotential over-fit of the discriminator network to the labels. A batchsize of 10 was used in the training.

Once all the fields-of-view have passed through the network 10, thewhole slide images are stitched together using the Fiji Grid/Collectionstitching plugin (see, e.g., Schindelin, J. et al. Fiji: an open-sourceplatform for biological-image analysis. Nat. Methods 9, 676-682 (2012),which is incorporated herein by reference). This plugin calculates theexact overlap between each tile and linearly blends them into a singlelarge image. Overall, the inference and stitching took ˜5 minutes and 30seconds, respectively, per cm² and can be substantially improved usinghardware and software advancements. Before being shown to thepathologists, sections which are out of focus or have major aberrations(due to e.g., dust particles) in either the auto-fluorescence orbright-field images are cropped out. Finally, the images were exportedto the Zoomify format (designed to enable viewing of large images usinga standard web browser; http://zoomify.com/) and uploaded to theGIGAmacro website (https://viewer.gigamacro.com/) for easy access andviewing by the pathologists.

Implementation Details

The other implementation details, including the number of trainedpatches, the number of epochs and the training times are shown in Table5 below. The digital/virtual staining deep neural network 10 wasimplemented using Python version 3.5.0. The GAN was implemented usingTensorFlow framework version 1.4.0. Other python libraries used were os,time, tqdm, the Python Imaging Library (PIL), SciPy, glob, ops, sys, andnumpy. The software was implemented on a desktop computer with a Corei7-7700K CPU @ 4.2 GHz (Intel) and 64 GB of RAM, running a Windows 10operating system (Microsoft). The network training and testing wereperformed using dual GeForce® GTX 1080Ti GPUs (NVidia).

TABLE 5 # of Training training # of time Virtual staining networkpatches epochs (hours) Salivary gland (H&E) 2768 26 13.046 Thyroid (H&E)8336 8 12.445 Thyroid (H&E, transfer learning) 8336 4 7.107 Liver(Masson's Trichrome) 3840 26 18.384 Lung (Masson's Trichrome) 9162 1016.602 Kidney (Jones stain) 4905 8 7.16 Liver (Masson's Trichrome, WSI)211475 3 39.64 Kidney (Jones stain, WSI) 59344 14 57.05 Ovary 1 4738 8437.21 Ovary 2 11123 14 37.41 Salivary Gland - 1 4417 65 24.61 SalivaryGland - 2 2652 90 23.9 Salivary Gland - 3 13262 24 30.58 Breast 67188 424.85 Skin 2566 124 27.02 Skin (DAPI + CY5) 2566 124 29.62 Prostate 677472 30.27

While embodiments of the present invention have been shown anddescribed, various modifications may be made without departing from thescope of the present invention. The invention, therefore, should not belimited, except to the following claims, and their equivalents.

What is claimed is:
 1. A method of generating a digitally stainedmicroscopic image of a label-free sample comprising: providing atrained, deep neural network that is executed by image processingsoftware using one or more processors of a computing device, wherein thetrained, deep neural network is trained with a plurality of matchedchemically stained images or image patches and their correspondingfluorescence images or image patches of the same sample; obtaining oneor more fluorescence images of the sample using a fluorescencemicroscope and one or more excitation light sources, wherein fluorescentlight is emitted from endogenous fluorophores or other endogenousemitters of frequency-shifted light within the sample; inputting the oneor more fluorescence images of the sample to the trained, deep neuralnetwork; and the trained, deep neural network outputting the digitallystained microscopic image of the sample that is substantially equivalentto a corresponding brightfield image of the same sample that has beenchemically stained.
 2. The method of claim 1, wherein the trained, deepneural network comprises a convolutional neural network.
 3. The methodof claim 1, wherein the deep neural network is trained using aGenerative Adversarial Network (GAN) model.
 4. The method of claim 1,wherein the sample is further labeled with one or more exogenousfluorescent labels or other exogenous emitters of light.
 5. The methodof claim 1, wherein the deep neural network is trained using a generatornetwork configured to learn statistical transformation between thematched chemically stained and fluorescence images or image patches ofthe same sample and a discriminator network configured to discriminatebetween a ground truth chemically stained image of the sample and theoutputted digitally stained microscopic image of the sample.
 6. Themethod of claim 1, wherein the sample comprises mammalian tissue, planttissue, cells, pathogens, biological fluid smears, or other objects ofinterest.
 7. The method of claim 1, wherein the deep neural network istrained with samples of the same type as the sample type of the obtainedone or more fluorescence image(s).
 8. The method of claim 1, wherein thetrained, deep neural network outputs a digitally stained microscopicimage in less than one second of inputting the one or more fluorescenceimage(s).
 9. The method of claim 1, wherein the sample comprises anon-fixed tissue sample.
 10. The method of claim 1, wherein the samplecomprises a fixed tissue sample.
 11. The method of claim 10, wherein thefixed tissue sample is embedded in paraffin.
 12. The method of claim 1,wherein the sample comprises a fresh tissue sample.
 13. The method ofclaim 1, wherein the sample comprises tissue imaged in vivo.
 14. Themethod of claim 1, wherein the excitation light source emitsultra-violet or near ultra-violet light.
 15. The method of claim 1,wherein the one or more fluorescence images are obtained at a filteredemission band or emission wavelength range using one or more filters ofa filter set.
 16. The method of claim 15, wherein a plurality of filtersare used to capture a plurality of fluorescence images which are inputto the trained, deep neural network.
 17. The method of claim 16, whereinthe plurality of fluorescence images are obtained by multiple excitationlight sources emitting light at different wavelengths or wavelengthbands.
 18. The method of claim 1, wherein the one or more fluorescenceimages is/are subject to one or more linear or non-linear imagepre-processing operations selected from contrast enhancement, contrastreversal, image filtering prior to being input to the trained, deepneural network.
 19. The method of claim 17, wherein the one or morefluorescence images and one or more pre-processed images are inputtogether into the trained, deep neural network.
 20. The method of claim1, wherein the plurality of matched chemically stained and fluorescenceimages or image patches of the same sample are subject to registrationduring training, comprising a global registration process that correctsfor rotation and a subsequent local registration process that matcheslocal features found in the matched chemically stained and fluorescenceimages.
 21. The method of claim 1, wherein the trained, deep neuralnetwork is trained using one or more GPUs or ASICs.
 22. The method ofclaim 1, wherein the trained, deep neural network is executed using oneor more GPUs or ASICs.
 23. The method of claim 1, wherein the digitallystained microscopic image of the sample is output in real time or nearreal time after obtaining the one or more fluorescence images of thesample.
 24. The method of claim 23, wherein the trained, deep neuralnetwork is trained for a second tissue/stain combination using initialneural network weights and biases from a first tissue/stain combinationwhich are optimized for the second tissue/stain combination usingtransfer learning.
 25. The method of claim 23, wherein the trained, deepneural network is trained for multiple tissue/stain combinations. 26.The method of claim 23, wherein the trained, deep neural network istrained for more than one chemical stain type for a given tissue type.27. A method of generating a digitally stained microscopic image of alabel-free sample comprising: providing a trained, deep neural networkthat is executed by image processing software using one or moreprocessors of a computing device, wherein the trained, deep neuralnetwork is trained with a plurality of matched chemically stained imagesor image patches and their corresponding fluorescence images or imagepatches of the same sample; obtaining a first fluorescence image of thesample using a fluorescence microscope and wherein fluorescent light ata first wavelength or wavelength range is emitted from endogenousfluorophores or other endogenous emitters of frequency-shifted lightwithin the sample; obtaining a second fluorescence image of the sampleusing a fluorescence microscope and wherein fluorescent light at asecond wavelength or wavelength range is emitted from endogenousfluorophores or other endogenous emitters of frequency-shifted lightwithin the sample inputting the first and second fluorescence images ofthe sample to the trained, deep neural network; and the trained, deepneural network outputting the digitally stained microscopic image of thesample that is substantially equivalent to a corresponding brightfieldimage of the same sample that has been chemically stained.
 28. Themethod of claim 27, wherein the first fluorescence image and the secondfluorescence image are obtained using different resolutions.
 29. Themethod of claim 27, wherein the sample comprises tissue, cells,pathogens, biological fluid smears, or other objects of interest.
 30. Asystem for generating digitally stained microscopic images of achemically unstained sample comprising: a computing device having imageprocessing software executed thereon or thereby, the image processingsoftware comprising a trained, deep neural network that is executedusing one or more processors of the computing device, wherein thetrained, deep neural network is trained with a plurality of matchedchemically stained images or image patches and their correspondingfluorescence images or image patches of the same sample, the imageprocessing software configured to receive one or more fluorescenceimage(s) of the sample and output the digitally stained microscopicimage of the sample that is substantially equivalent to a correspondingbrightfield image of the same sample that has been chemically stained.31. The system of claim 30, wherein the trained, deep neural networkcomprises a convolutional neural network.
 32. The system of claim 31,wherein the trained, deep neural network is trained using a GenerativeAdversarial Network (GAN) model.
 33. The system of claim 30, furthercomprising a fluorescent microscope configured to obtain the one or morefluorescence image(s) of the sample.
 34. The system of claim 33, furthercomprising a plurality of filters wherein a plurality of fluorescenceimages are obtained using different filters.
 35. The system of claim 33,wherein the fluorescent microscope comprises multiple excitation lightsources emitting light at different wavelengths or wavelength bands. 36.The system of claim 32, wherein the GAN model is trained using agenerator network configured to learn statistical transformation betweenthe matched chemically stained and fluorescence images or image patchesof the same sample and a discriminator network configured todiscriminate between a ground truth chemically stained image of the samesample and the outputted digitally stained microscopic image of thesample.
 37. The system of claim 32, wherein the GAN model is trainedwith a sample of the same sample type as the sample of the obtained oneor more fluorescence image(s).
 38. The system of claim 30, wherein thetrained, deep neural network outputs a digitally stained microscopicimage in less than one second of inputting the one or more fluorescenceimage(s).
 39. The system of claim 30, wherein an excitation light sourceof the fluorescent microscope emits ultra-violet or near ultra-violetlight.
 40. The system of claim 33, wherein the one or more fluorescenceimage(s) is/are obtained at a filtered emission band or emissionwavelength range using a filter set.
 41. The system of claim 40, whereinthe filter set comprises one of a plurality of filters configured foruse with the fluorescence microscope.